The tangiBook: A Tangible Display System for Direct Interaction with Virtual Surfaces

In this paper we introduce the tangiBook, a tangible display system that allows realistic simulation and direct interaction with virtual surfaces. The tangiBook is based on an off-the-shelf laptop computer that incorporates an accelerometer and a webcam as standard equipment. Custom software allows the orientation of the laptop screen and the position of the observer to be tracked in real-time. Using this information, realistic images of surfaces with complex textures and material properties, illuminated by environment-mapped lighting, are rendered to the screen at interactive rates. Tilting the laptop or moving in front of the screen produces realistic changes in surface lighting and material appearance. Thus the tangiBook allows virtual surfaces to be observed and manipulated as naturally as real ones, with the added benefit that the material properties of the surfaces can be changed in real-time. We demonstrate the potential utility of the tangiBook in three application areas: material perception research, computer-aided appearance design, and enhanced access to collections in digital libraries and museums. With its unique capabilities and commodity roots, the tangiBook can be broadly deployed and should be useful in a wide range of applications. Introduction Advances in computer graphics technology have provided the capabilities for realistic image synthesis, allowing images of virtual objects to be rendered that are indistinguishable from photographs of real-world objects [1]. The ability to create realistic renderings of virtual objects has proven to be valuable in a variety of application domains including the film and game industries, computer-aided design and manufacturing, medical imaging and digital libraries and museums. With standard computer graphics systems, a user is a step removed visually and physically from the virtual objects they are interacting with. The screen acts as a window separating the real and virtual worlds, and the user does not interact directly with the virtual objects, but instead uses indirect means, such as a mouse or keyboard, to manipulate the objects or shift the point of view. Our objective is to remove these barriers between the real and virtual worlds by developing a tangible display system, which we call the tangiBook, that gives users much of the same experience they would have if the virtual object were situated in a real environment (see Figure 1). Instead of providing a view into a virtual world, this system presents the virtual object, situated in the real world, at the physical location of the display. It supports natural interaction with the object, through direct manipulation of the object’s orientation by rotating the display, and dynamic viewpoint changes through observer tracking. The tangiBook is based on an off-the-shelf laptop computer that contains a triaxial accelerometer and a webcam as standard components. Through custom software that integrates these devices, we are able to actively sense the orientation of the laptop display and dynamically track the observer’s viewpoint with camera-based head-tracking. This information is used to drive a physically-based rendering algorithm that generates an accurately oriented and realistically shaded view of a virtual surface to the laptop’s display. The custom rendering system also allows the user to change the material properties of the rendered virtual surface in real-time, a capability that allows the tangiBook to be used in psychophysical experiments and computer-aided appearance design applications. Related Work There has been a strong interest in developing natural interfaces for viewing virtual objects and scenes since Sutherland and colleague’s pioneering work on the design of a see-through head-mounted display (HMD) and 3D input wand in the 1960’s and 1970’s [2, 3]. Significant advances have been made in both display systems [4, 5, 6], and 3D input devices [7, 8, 9] for virtual environments. The focus in these systems is on immersing a user in a virtual world delivered through a HMD. An alternative approach is represented by the CAVE system [10] that surrounds a user with projection screens on which virtual content is displayed. Further innovations in this projection-based approach include the shaderLamps and iLamps systems [11,12, 13] that project pre-warped images on real three-dimensional objects to create physical objects whose appearance can be changed under computer control. Bimber and colleagues have also been early innovators in this area (see [14] for a comprehensive review). A third major approach to bridging real and virtual worlds incorporates spatially aware displays and tangible interfaces. The Chameleon systems [15,16] coupled 6 DOF trackers with computer displays to create spatially aware display systems whose content changes depending on its position and context. The Virtual Mirror system [17] took the spatially aware display concept a step further by incorporating a video camera pointed toward the user holding the display to create the impression of a real mirror. The device also allowed interactive viewing of reflective daguerreotype images [18]. A significant feature of these systems is the capability to grasp and directly manipulate the display to change the viewpoint on the virtual environment. The virtue of tangible interfaces such as these is that affordances of the object (in these cases lifting, tilting, and rotating the display) support rich and natural modes of interaction with the virtual world. System Design Our objective in designing the tangiBook was to create a lowcost-of-entry tangible display system for interacting with virtual surfaces. The tangiBook is based on an off-the-shelf laptop computer (Apple MacBook Pro) that incorporates all the components necessary to create a tangible display system: an LCD display, an accelerometer, and a webcam. The accelerometer is used to estimate the orientation of the display, and the camera provides a means to estimate the observer’s viewing position with computer vision-based head tracking. This information is used within a custom 3D shader to dynamically render a realistically shaded view of the virtual surface to the laptop’s display. By integrating these components in a custom software system, virtual surfaces can be observed and manipulated in the same manner as real surfaces, such that tilting the laptop or moving in front of the screen produces realistic changes in surface appearance. An additional capability is that the material properties of the virtual surface can be changed in real-time. In the following sections we describe the implementation of each of these capabilities.